DDF Library: Enabling functional programming in a task-based model

In recent years, the areas of High-Performance Computing (HPC) and massive data processing (also know as Big Data) have been in a convergence course, since they tend to be deployed on similar hardware. HPC systems historically performed well regular, matrix-based computations; other hand, Data problems often excelled fine-grained, parallel workloads. While programming is mostly task-based, like COMPSs, popular environments, Spark, adopt functional paradigm. A careful analysis shows that there are pros cons both approaches, integrating them may yield interesting results. With reasoning mind, we developed DDF, an API library for COMPSs allows developers use techniques while using environment. DDF has functional-based interface, many Science tools, us dynamic evaluation adapt task execution run time. It brings some qualities programming, making it easier application domain experts write Analysis jobs. this article discuss evaluate impact used its implementation allow more efficient execution. addition, present performance comparison with Spark several patterns. The results show each technique significantly impacts performance, allowing outperform cases.